Skip to content

test: fix the fleet integration suite's flaky waits - #141

Merged
outofcoffee merged 2 commits into
mainfrom
fix/fleet-metrics-race
Sep 1, 2026
Merged

test: fix the fleet integration suite's flaky waits#141
outofcoffee merged 2 commits into
mainfrom
fix/fleet-metrics-race

Conversation

@outofcoffee

@outofcoffee outofcoffee commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Fixes both flakes in the fleet-integration job: metrics assertions that raced the daemon's sampler, and docker compose failures that were discarded rather than reported.

Summary

  • Add wait_for_tokens and use it in test_metrics, which asserted on the engine's counters as soon as the node reported running.
  • Route the compose lifecycle commands through a helper that stays quiet on success and prints the captured output and exit code on failure.
  • Retry a node restart while waiting for it, instead of issuing one start and assuming it took; dump container state and logs when a wait times out.
  • Read fleet status into a variable in wait_for_fleet rather than piping it to grep -q.

Implementation details

The metrics race. running means the engine process is alive (internal/daemon/supervisor.go) and nothing more. The counters take a different path: Daemon.Metrics reports the reading its background sampler last took rather than scraping when asked, and SampleActivity retries every second until the first reading lands. The fake engine is Imposter exec'd by the shim, so it also needs a moment to bind its port. That leaves a window where the state is running, the resource bars render because they come from the node rather than the engine, and the counters do not exist yet — a run landing in it fails prompt tokens and 4096 while RAM passes on the same output, which is what CI showed.

Waiting on readiness would not close it: sampleOnce runs before checkReadyOnce in the same loop iteration, so an engine that comes up between the two reads as ready with no sample behind it until the next tick.

The silent failures. Every docker compose call discarded stdout, stderr and exit status. Two CI failures came out of that: a start laptop that left the node refusing connections for ninety seconds with no explanation, and a bring-up that failed ten seconds in and printed nothing but Tearing down.... The restart is now retried while waiting, so a start that loses the race for its published host port no longer strands the node for the rest of the run.

The pipeline. Under pipefail, a grep -q that matches exits immediately and the writer can take a SIGPIPE, making the pipeline report 141; the ! then reads that as "nothing unreachable", the opposite of what was found. Today's three-line output never fills the pipe buffer so it does not trigger, but it is a wait helper that can report ready when it is not.

Verified by running the suite against Docker locally (all assertions pass), and by exercising restart_node with injected start failures to confirm it retries and recovers.

A node reads `running` as soon as the engine process is alive, but the
token counters come from the daemon's background sampler, which retries
about once a second until its first reading lands — and the fake engine
needs a moment to answer at all. The metrics assertions ran inside that
window, so CI saw resource bars (taken from the node) with no counters
beside them. Poll for the counters instead of the state.
@outofcoffee outofcoffee added the bug Something isn't working label Sep 1, 2026
Every docker compose call sent its output and exit status to /dev/null,
so a failure explained nothing. A bring-up that could not build left only
"Tearing down..." and an exit code behind, and a failed `start laptop`
became ninety seconds of connection-refused with no clue why — one
attempt was made and never retried, so a node that lost the race for its
published host port stayed down for the rest of the run.

Route the lifecycle commands through a helper that stays quiet on success
and reports the captured output on failure, retry the restart while
waiting for it, and dump container state and logs when a wait times out.

Also read `fleet status` into a variable rather than piping it to grep:
under pipefail a matching `grep -q` can exit first and leave the pipeline
reporting the writer's SIGPIPE, which the caller reads as "nothing
unreachable" — the opposite of what was found.
@outofcoffee outofcoffee changed the title test: wait for the engine counters before asserting on them test: fix the fleet integration suite's flaky waits Sep 1, 2026
@outofcoffee
outofcoffee merged commit a376f7c into main Sep 1, 2026
2 checks passed
@outofcoffee
outofcoffee deleted the fix/fleet-metrics-race branch September 1, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant